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Capability  Test  Design  &  Analysis 

Objectives 

1.  Discuss  the  concept  of  conducting  a  capability 
evaluation  strategy  refinement  process  for  testing  in 
a  joint  environment  (TlJE) 

2.  Review  the  methods  and  processes  for  an  evaluation 
strategy  refinement  process 

3.  Review  potential  design  of  experiment  techniques  for 
large  number  of  factors 

4.  Review  tools  and  techniques  for  an  evaluation 
strategy  refinement  process 

5.  Step  through  a  “case  study”  example  of  an 
evaluation  strategy  refinement  process 

6.  Review  potential  issues  and  insights 
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Exploratory  Analysis  Purpose 


Factor 


Factor  1 


Factor  2 


Factors 


Factor  4 


Factors 


Factors 


Factor? 


Factors 


Factors 


Factor  10 


Factor  Range 


1m  In 


'2m  In 


Xmln 


^4mln 


^6mln 


tmln 


'?mln 


^Xmln 


'•min 


MOmln 


Imai 


'2m 


Xm 


'4m 


'6m 


•m 


'7m 


'Xm 


'•mai 


'lOmai 


Factor  ranges  of 
interest 


Refined  factor 
ranges  of 
interest 


Factors 

eliminated  from 
further  t 
consideration  i 


initiai  JMe  Factor  Test  Space 


Factor  Factor  Range 

jta,  »  X, 


Factor  10 


MOmln 


10m  a  I 


Refined  JMe  Factor  Test  Space 


Purpose  is  three-fold: 

-  To  explore  a  wide  range  of  possible  factors  and  levels  that  might  affect  joint 
mission  effectiveness  (JMe),  referred  to  as  the  initial  JMe  factor  test  space; 

-  To  identify  those  combinations  of  factors  that  have  the  greatest  impact  on  JMe, 
referred  to  as  the  refined  JMe  factor  test  space;  and 

-  To  recommend  potential  factor  combinations  of  interest  from  the  refined  JMe 
factor  test  space  for  subsequent  test  events,  referred  to  as  potential  test  trial 
sets. 
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CTM  0.3. 3.2:  Develop/Refine 
Exploratory  Analysis  Plan 
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CTM  0.3.3. 3:  Execute  Exploratory 

Data  Collection 
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CTM  0.3. 3.4:  Conduct  Exploratory 

Analysis 
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CTM  0. 3.3.5:  Synthesize 
Exploratory  Analysis  Results 
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to  achieve 

Desired  Effect  I  Desired  Effect  Name 
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Develop  Evaluation  Framework 
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Factor  Capability  Crosswalk 
SME  Estimates 
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step  1 :  “Quick  Look”  Analysis 


Dependent  Variable:  Number  Threat  Kills 
Independent  Variables:  17  factors  (decision  &  conditional) 

DOE:  Resolution  III  Fractional  Factorial  (80  trials,  20  runs  each) 
Analysis  tools:  Stepwise  Regression  Model 
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Findings: 

•  14  factors  significant,  3  factors  not  significant 

•  Adverse  weather  factor  non-intuitive 
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Step  2:  Main  Effects  Analysis 

Dependent  Variable:  Number  Threat  Kills 
Independent  Variables:  14  factors  (decision  &  conditional) 

DOE:  Resolution  V  Fractional  Factorial  (2304  trials,  3  runs  each) 
Analysis  tools:  Classification  and  Regression  Tree  (CART)  partitioning 


item@ite.osd.mil 


Findings: 

•  Stressing  factors  (Blue  speed,  ROZ  size,  ROZ  type) 


Step  2:  Main  Effects  Analysis 

Dependent  Variable:  Number  Threat  Kills 

Independent  Variables:  14  factors  (decision  &  conditional) 

DOE:  Resolution  V  Fractional  Factorial  (2304  trials,  3  runs  each) 
Analysis  tools:  Stepwise  Regression  Model 
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Findings: 

•  10  factors  significant 

•  Blue  fires  no  longer  significant 

•  Adverse  weather  factor  intuitive 
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SME 


Step  2:  Main  Effects  &  Two-way  Interaction 

Dependent  Variable:  Number  Threat  Kills 

Independent  Variables:  14  factors  (decision  &  conditional) 

DOE:  Resolution  V  Fractional  Factorial  (2304  trials,  3  runs  each) 
Analysis  tools:  Stepwise  Regression  Model 
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Findings: 

•  6  main  effect  factors  significant 

•  5  additional  factors  significant  in  two-way  interactions 

•  Blue  fires  &  adverse  weather  part  of  two-way  interactions 
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step  3:  Aggregated  Conditional  Factors 

Dependent  Variable:  Number  Threat  Kills 
Independent  Variables:  10  factors  (9  decision,  1  conditional) 

DOE:  Resolution  V  Fractional  Factorial  (128  trials,  3  runs  each) 
Analysis  tools:  Classification  and  Regression  Tree  (CART)  partitioning 
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/lean  73.03125  4.1947918  21.9375 

5tdDev  18.767653 


Blue  1 

»eetl^=2 

Count 

32 

LogWortli 

Difference 

Mean 

96.53125 

2.9392673 

20.4375 

Std  Dev 

20.130196 

Blue  S|jeecl<2 

Count  32 

LogWortli 

ifference 

Mean 

100.90625 

4.2290919 

20.4375 

Std  Dev 

17.43256 

Blue  Speed>=2 

Count 

32 

LogWoilh 

Difference 

Mean 

139.03125 

2.7937483 

24.0625 

Std  Dev 

24.225399 

GIG?(Off> 

GIG?(Ont 

Count  16 

Count 

16 

Mean  62.0625 

Mean 

84 

Std  Dev  10.266247 

Std  Dev  19.127641 

GIG  Off) 

GIG?(On) 

Count 

16 

Count  1 6 

LogWertli 

Difference 

Mean  ■ 

■  -.3125 

Mean  106.75 

4.2011102 

28 

Std  Dev  1 

284748 

Std  Dev  17.774514 

CFF  Type  of  Decision(A) 

CFF  Type  of  ^cision(C) 

CFF  Type  of  Decision(A) 

CFF  TyiJe  of  DecisionfC 

Count 

16 

Count 

S 

Count 

16  LogWorth 

Difference 

Count 

16 

Mean 

90.6875 

Mean 

111.12. 

Mean 

127  2.9132009 

34 

Mean 

151 .0625 

Std  Dev 

14.286211 

Std  Dev 

14.183911 

Std  Dev 

24.144012 

Std  Dev 

17.920077 

ROZ  TypefR) 

ROZ  Tyi)e(P) 

Count 

8 

Count 

8 

Mean 

92.75 

Mean 

120.75 

Std  Dev 

9.6473534 

Std  Dev  11.658841 

1  ROZ  Tyi>e(R) 

ROZ  TypefP) 

"ount 

8 

Count  8 

Ml.  -■ 

110 

Mean  144 

Std  Dt 

14.696938 

Std  Dev  19.302109 
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step  3:  Aggregated  Conditional  Factors  Results 

Sorted  Parameter  Estimates 


Term 

Esti  mate 

Std  Error 

t  Ratio 

(5  Factor  Least  Stressing  Levels) 

17.E937E 

1.12Z227 

15.63 

Blue  Speed 

30.S125 

2.244^55 

13.73 

(Blue  Fires^l  .5)*R0Z  Type[P! 

-1E.S125 

2.244455 

-7.05 

ROZType[P] 

e.37E 

1.122227 

5.63 

CFF  Type  of  BecisianlAJ 

-6.2eeS7E 

1.122227 

-5.61 

GIG?[0f1J 

-5.&0625 

1.122227 

-5.26 

MultiuleTrackersZfNaT 

-5.59375 

1.122227 

-4.93 

Prob>|t| 

<.Q00r 

<(>001* 

<0001* 

<0001* 

<0001* 

<0001* 

<0001* 


Dimension 

Sub  Dimension 

Priority/Factor 

Levels 

Factor  Type 

System  of 
Systems 

Materiel 

1.  Blue  Speed 

1/2  (Multiplier) 

Continuous 

2.  Blue  Fires 

1/2  (Multiplier) 

Continuous 

5.  Global  Information  Grid  (GIG)? 

Yes/No 

Categorical 

3.  Restricted  Op.  Zone  Type 

Restri  cti  ve/Pe  r  m  i  ss  i  ve 

Categorical 

4.  Call  For  Fire  type  Decision 

A/C 

Categorical 

6.  Multiple  Trackers? 

Yes/No 

Categorical 

Do 

ctrine 

Most  Stress  1 .  ROZ  Size 

1  (Multiplier) 

Continuous 

Most  Stress  2.  ROZ  Expiration 

No 

Categorical 

Most  Stress  3.  Multi-Service  Wait 
time 

2  (Additional  Time 
Increment) 

Continuous 

Condition 

Eivir 

mmental 

Most  Stress  4.  Adverse  Weather? 

Yes 

Categorical 

Most  Stress  5.  Civilian  Zone? 

Yes  (Size  =  40) 

Categorical 
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Findings: 

•  One  aggregated  condition  factor  significant 

•  Six  decisional  factors  significant 


step  1:  Measures  Relationship  Table 


Green:  Direct  relationship 
Red:  Indirect  relationship 
No  color:  No  relationship 


M  M 
M  M 
0  0 
E  E 


T  T  T 
M  M  M 
0  0  0 
P  P  P 


M  M 
0  0 


S 

A 


S 

A 


GIG 

Off  or  On 

1 

1 

1 

1 

1 

0 

1 

1 

1 

1 

1 

1 

1 

0 

1 

0 

1 

0 

1 

0 

1 

0 

1 

Ellue  Speed 

1  or  2 

1 

1 

1 

0 

0 

1 

0 

1 

0 

0 

1 

0 

0 

1 

0 

1 

0 

1 

0 

0 

0 

0 

0 

Blue  Monitor 

1  or  2 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

Blue  Fires 

1  or  2 

1 

1 

0 

-1 

0 

1 

-1 

0 

0 

0 

0 

1 

1 

0 

0 

1 

0 

0 

Civilian  Zone 

Off  or  On 

0 

0 

1 

0 

1 

1 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

1 

0 

0 

1 

1 

Civilian  Zone  Size 

Small  or  Large 

-1 

-1 

-1 

0 

0 

-1 

0 

1 

0 

0 

1 

0 

0 

1 

0 

1 

0 

1 

0 

0 

0 

0 

0 

Multiple  Trackers 

No  or  Yes 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

Adverse  Weather 

No  or  Yes 

1 

1 

1 

1 

0 

0 

1 

1 

1 

0 

1 

1 

0 

0 

1 

0 

1 

0 

1 

0 

1 

0 

0 

Expedited  CFF 

No  or  Yes 

1 

1 

0 

1 

0 

0 

0 

0 

1 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

CFF  Type  of  Decision 

Closest  or  Available 

-1 

-1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

Expedite  Move 

No  or  Yes 

1 

1 

1 

-1 

0 

1 

0 

1 

oJ 

lo 

0 

0 

ROZ  Type 

Restrictive  or  Permissive 

1 

1 

1 

1 

1 

1 

-1 

1 

1 

1 

1 

1 

1 

1 

mI 

pn 

s 

1 

ROZ  Size 

1  or  2 

-1 

-1 

-1 

0 

0 

-1 

1 

1 

0 

0 

1 

0 

0 

1 

mI 

pn 

pr 

0 

0 

0 

Multi-service  Wait  time 

1  or  2 

0 

0 

0 

1 

0 

1 

0 

0 

0 

0 

0 

0 

0 

oj 

pr 

0 

1 

0 

0 

0 

ROZ  Expiration 

No  or  Yes 

1 

1 

-1 

-1 

-1 

-1 

-1 

1 

1 

1 

1 

1 

1 

ROZ  Slack  Time 

1  or  2 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

1 

Civilian  Zone  Location 

1  or  2 

0 

0 

-1 

0 

-1 

-1 

-1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

1 

0 

0 

1 

1 

Case  Study 
Example 


M  M  M  M  M  M 


1  1  2 
T  T  T 


2 

T 


M  M  M  M 


1  1 
S  S 


2 

S 


T  T 
1  1 
S  S 


T 

2 

S 


T 

2 

S 


T 

3 

S 


Findings  (Direct  and  Indirect  relationships): 

•  Direct  relationship  for  all  measures:  Blue  Monitor,  Multiple  Trackers,  ROZ  Slack  Time 

•  Indirect  relationship  for  MMOEs  and  TMOPs/MOSAs:  CFF  Type  of  Decision,  ROZ  Expiration 

•  Direct  relationship  across  both  MMOEs 


MMOE  -  Mission  Measure  of  Effectiveness  MOSA  -  Measure  of  System  of  Systems  Attributes 
TMOP  -  Task  Measure  of  Performance 
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Insights  into  Expioratory  Anaiysis 

•  Resolution  V  DOE  needed  for  assessing  two-way  interactions 

-  Resolution  III  does  not  confound  main  effects  with  one  another,  but  does 
confound  main  effects  with  2-factor  interactions 

-  Resolution  V  does  not  confound  main  effects  and  2-factor  interactions,  but 
confounds  main  with  4-factor  and  2-factor  with  3-factor 

•  Factor  prioritization  is  an  iterative  process 

-  Initial  DOE  and  data  farming  may  provide  first  insights  into  significant  measures 

-  May  require  further  exploration  to  validate  initial  findings 

-  May  differ  across  multiple  measures  and  require  retaining  uncertain  factors  in 
the  second  design 

-  Iterative  farming  can  provide  additional  prioritization  of  factors 

•  Factors  with  more  than  two  discrete  levels  requires  additional  farming 
to  assess  their  impact 

-  Requires  crossing  with  additional  factors 

-  May  wish  to  assume  two  levels  for  initial  design 

•  Multiple  measures  (dependent  variables)  adds  significant  complexity  to 
determining  factors  with  highest  impact 

-  Requires  evaluation  of  factors  across  measures 

-  Constructing  relationship  tables  provides  insights  on  measure  impacts 
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Summary 

•  Exploratory  analysis  requires  an  iterative 
process  for  prioritizing  factors 

•  Factors  can  be  analyzed  across  multiple 
dependent  variables  (measures) 

•  Automated  tools  for  DOE  and  modeling  can 
help  to  simplify  the  exploratory  analysis 
process 

•  Non-materiel  factors  can  be  equally  important 
to  testing  a  System  of  Systems 
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Abstract 


The  Joint  Test  and  Evaluation  Methodology  (JTEM)  project  has  been  collaborating 
with  various  government  organizations  and  academia  to  develop  enhanced  Design  of 
Experiment  (DOE)  modeling  and  analysis  approaches  for  Testing  in  a  Joint 
Environment  (TlJE).  This  paper  discusses  the  applied  research  that  has  been 
conducted  in  this  area  over  the  past  three  years,  as  well  as  its  application  to  JTEM 
test  events.  Discoveries  involving  enhanced  data  farming  techniques  and  technology 
applications  have  proven  to  be  catalysts  for  test  and  evaluation  of  complex  adaptive 
systems.  Hybrid  DOE  models  for  large  factor  test  designs  (e.g.,  Fractional  Factorial 
Controlled  Sequential  Bifurcation,  Resolution  Five  Fractional  Factorial,  Nearly 
Orthogonal  Latin  Hypercube)  have  demonstrated  success  in  refining  robust  Joint  test 
spaces.  Innovative  application  of  analytical  models  and  methodologies  (e.g.. 
Advanced  Response  Surface  Methodology,  Classification  and  Regression  Tree)  have 
improved  our  ability  to  analyze  Critical  Capability  Issues  (CCI)  involving  multiple 
responses.  Agent  based  model  simulation  prototypes  (e.g..  Tester,  MANA, 
Pythagoras)  have  been  modified  and/or  developed  by  our  academic  and  government 
partners  to  enable  enhanced  test  design  and  evaluation  of  capabilities  in  a  Joint 
environment.  Proof  of  concept  efforts  in  this  collaboration  has  included  International 
Data  Farming  Workshop  (IDFW)  events,  where  various  techniques  and  tools  have 
been  explored  for  use  in  Testing  in  a  Joint  Environment  (TlJE).  Key  research 
techniques  and  selected  results  are  presented  in  the  context  of  a  use  case  that  is 
based  upon  JTEM  test  events. 
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